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Abstract 

This  paper  describes  a  project  undertaken  to  explore  programming  physical  operations  on 
complex  flexible  objects.  Uncertainty  about  the  exact  state  of  the  object  makes  it  impossible  to 
precisely  specify  the  actions  to  be  performed  at  the  time  the  program  is  written.  Furthermore, 
the  detailed  consequences  of  manipulations  on  flexible  objects  cannot  be  determined  before  the 
action  is  performed.  Lacking  precise  specifications,  the  programmer  must  abstract  the  essential 
properties  of  objects  and  actions. 

In  an  effort  to  study  manipulation  of  flexible  objects,  a  system  to  tie  knots  in  rope  with  a 
robot  arm  was  developed.  The  system  includes  an  extensible,  graph  representation  for  knots,  a 
vision  system  that  binds  the  contour  of  a  physical  rope  to  an  abstract  description,  and  a  knot 
tying  language  based  on  parametric  motion  commands.  Knots  of  modest  complexity,  such  as  a 
bowline  or  figure  eight,  can  be  tied  in  a  variety  of  ropes  with  minimal  constraints  on  the  initial 
configuration  of  the  rope.  The  work  highlights  the  importance  of  software  engineering  princi¬ 
ples  and  a  good  programming  environment  for  robot  program  development.  ^ 
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1.  Introduction 

Most  work  in  robotics  has  been  directed  towards  rigid  objects  with  simple,  regular  shapes. 
Object  geometry  is  usually  very  predictable,  varying  only  within  small  tolerances  from  one 
instance  of  an  object  to  another.  Conventional  robots  and  robot  programming  languages  rely 
on  this  regularity  by  imposing  a  rigid  structure  on  the  world.  The  robot  must  know  the  exact 
form  and  position  of  objects  in  the  environment  and  the  precise  actions  to  be  performed. 

Many  objects  that  we  deal  with  everyday  are  irregular,  pliant,  and  have  highly  variable 
shapes.  Flexible  seals,  gaskets,  clips,  and  hoses  are  common  components  in  assemblies. 
Materials  such  as  rubber,  cloth,  paper,  wire,  or  rope  are  difficult  to  handle  and  impossible  to 
model  completely.  Conceptually  simple  manipulations  are  difficult  to  specify  and  simple 
actions  can  produce  complex  changes  to  the  objects.  In  this  environment,  robots  and  robot 
languages  that  depend  on  precise  information  and  fine  tolerances  are  difficult  to  program  and 
extremely  error  prone. 

This  paper  reports  on  an  experiment  undertaken  to  study  methods  for  programming  mani¬ 
pulations  of  flexible  objects.  The  task  chosen  was  tying  knots  in  rope.  Knots  provide  a  rich 
variety  of  well  defined  objects  whose  interwoven  structure  's  difficult  to  characterize.  Rope 
handling  is  difficult  to  control  because  movements  can  affect  the  shape  and  position  of  the  rope 
in  complex  and  unpredictable  ways.  As  a  consequence,  robot  programs  to  tie  knots  cannot 
depend  on  the  exact  structure  of  the  rope  and  must  rely  on  sensory  information  (Inoue,  1984). 

Our  goal  was  not  to  tie  a  single  knot,  but  to  devise  a  gramnaar  of  knots  sufficient  to  natur¬ 
ally  express  the  manipulations  needed  to  tie  many  different  knots.  The  grammar  was  to  be 
independent  of  the  device  used  manipulate  the  rope,  the  size  of  the  rope,  and  the  detailed 
configuration  of  the  rope.  We  also  intended  to  develop  a  system  to  convert  programs  written  in 
this  grammer  into  robot  manipulations  on  an  actual  rope. 

The  language  we  created  is  composed  of  parametric  command  specifications.  The  project 
illustrates  the  importance  of  software  engineering  principles  and  programming  environments 
for  building  layers  of  abstraction  in  robot  programming  languages.  The  experience  also 
emphasizes  the  interdependence  of  language,  action,  and  vision  for  complex  manipulation 
problems. 

2.  Problem  Definition  and  Solution  Requirements 

The  project  was  motivated  by  a  desire  to  study  programming  for  physicals  objects  whose 
exact  form  is  unknown  at  programming  time.  For  this  reason,  we  chose  to  avoid  techniques 
that  would  limit  the  rope’s  freedom  of  motion.  Special  purpose  fixtures  were  ruled  out  as  being 
to  restrictive.  The  rope  must  naturally  trail  and  twist  as  a  portion  is  grasped,  carried,  and 
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dropped. 

Methods  were  not  to  depend  on  metric  properties  of  the  rope.  Knot  tying  programs  were 
to  work  for  a  range  of  sizes,  stifEnesses,  and  initial  configurations.  From  the  programmer’s  per¬ 
spective  the  referent  need  not  even  be  a  knot.  The  language  should  have  descriptive  power  to 
characterize  the  threading  of  smooth  space  curves.  Road  layouts  and  script  handwriting  have  a 
woven  structure  that  is  similar  to  knots  and  could  potentially  be  described  in  the  language. 

A  single  length  of  rope  was  used.  The  rope  was  placed  on  a  table  with  no  other  objects  in 
the  workspace.  The  rope  was  viewed  from  a  camera  placed  above  the  table.  A  Puma  560 
robot  arm  was  used  to  carry  out  the  manipulations.  Figure  1  shows  the  robot  performing  an 
intermediate  step  in  a  knot  program. 

<INSERT  FIGURE  1  a^OUT  HERE> 

A  wide  variety  of  knots  i  an  be  tied  in  a  single  rope  with  one  arm  by  weaving  a  piece  of  the 
rope  over  and  under  sections  lying  on  the  table.  We  envisioned  our  task  as  the  identification  of 
constructs  for  specifying  die  interweaving  of  rope  and  the  determination  of  robust  methods  to 
achieve  them. 

3.  Object  Level  Programming 

The  approach  we  pursued  was  to  begin  by  designing  an  abstract  description  of  a  rope  and 
a  set  of  logical  operations  to  transform  the  rope  from  one  state  to  another.  Since  the  exact 
structure  of  the  rope  was  unknown,  the  description  could  not  depend  on  absolute  spatial  coordi¬ 
nates.  The  development  process  applied  the  methodology  of  abstract  data  types  to  operations 
on  physical  objects  (Liskov,  1986). 

An  abstract  data  type  presents  a  programmer  with  abstract  operations  on  an  object.  The 
specific  structure  of  the  object  and  the  details  of  how  operations  change  the  object  are  hidden 
from  view.  In  typical  applications,  the  data  type  maintains  an  internal  data  structure  that 
represents  the  state  of  the  data  abstraction.  Operations  have  a  deterministic  effect  on  the 
representation.  The  initial  state  of  the  representation  is  known  and  each  operation  changes  it  in 
a  fixed  and  predetermined  way. 

Indeterminacy  in  the  object  state  distinguishes  the  implementation  of  abstract  data  types 
for  physical  objects  from  their  normal  usage.  The  initial  state  of  the  object  cannot  be  known 
exactly.  Furtheimore,  actions  have  a  nondeterministic  effect  on  the  the  objects  The  state  of  a 
physical  object  must  be  inferred  from  sensory  data.  Perceptual  and  actions  systems  are  needed 
to  map  the  abstract  operations  onto  a  physical  instance  of  the  object.  The  interpretation  of 
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sensory  input  must  be  guided  by  the  data  type.  The  meaning  of  an  image,  for  example,  depends 
on  the  abstractions  defined  by  the  data  type.  The  data  type  may  or  may  not  maintain  an  inter¬ 
nal  representation  of  the  object  If  an  internal  representation  is  used,  then  the  data  type  must 
provide  an  interface  to  the  perceptual  system  to  allow  creation  and  modification  of  the  underly¬ 
ing  data  structures. 

The  value  of  programming  with  high-level  constructs  is  well  accepted  in  the  field  of  robot 
language  design.  Robot  languages  can  be  characterized  by  the  view  of  the  world  they  present 
to  the  programmer.  There  are  several  classification  schemes  that  define  hierarchical  taxo¬ 
nomies  of  robot  languages  (Lieberman,  1977,  Lozano- Perez,  1983,  Ambler,  1987,  Volz,  1988). 
These  taxonomies  can  be  used  not  only  to  classify  individual  languages,  but  can  also  be  viewed 
as  multiple  layers  on  which  a  single  system  can  be  constructed.  Each  successive  level  is  built 
from  the  set  of  primitives  provided  by  levels  below. 

Commercially  available  languages  require  programming  at  the  robot-level.  The  program¬ 
mer  must  directly  specify  movements  of  the  robot  Movement  primitives  give  the  position, 
orientation,  or  compliance  of  a  point  on  the  end-efiector.  Lower  levels  solve  the  inverse 
kinematics  necessary  to  determine  joint  positions  and  plan  trajectories  (Finkel,  1975,  Paul, 
1976,  Shimano,  1979,  Taylor,  1982,Geschke,  1983).  Experience  has  shown  that  robot  level 
programs  arc  difiBcult  to  write,  edit,  debug,  and  modify  (Gini,  1985).  Part  of  the  difficulty  is 
caused  by  the  unnaturalness  of  metric  definitions  of  three-dimensional  positions  and  orienta¬ 
tions.  More  fundamentally,  the  dependence  on  precise  information  limits  the  generality  and 
applicability  of  robot  programs.  When  objects  are  flexible,  when  object  shapes  arc  complex,  or 
when  motion  sequences  are  intricate,  it  is  difficult  to  obtain  precise  information  about  object 
location  and  to  enforce  fine  tolerances  on  motions. 

To  remedy  the  awkwardness  and  limitations  of  robot-level  programs,  researchers  have 
studied  methods  to  elevate  the  level  of  abstraction  in  robot  languages.  Object-level  languages 
allow  the  programmer  to  specify  geometric  constraints  on  object  positions  (Poplestone, 
1980,Poplestone,  1983,Latombc,  1985).  Each  statement  specifies  a  partial  constraint  on  the 
relationship  between  objects.  The  command 

’’place  face  1  of  object  B  against /ace  2  of  object  C" 

states  that  surfaces  on  two  objects  should  be  in  contact  If  face  1  znd  face  2  arc  planar,  then  the 
statement  means  that  the  surfaces  should  be  coplanar  with  surface  normals  pointing  in  oppos¬ 
ing  directions.  The  compiler  solves  a  system  of  constraints  given  by  a  set  of  statements  using 
internal  object  models.  Object-level  programs  are  translated  into  robot-level  programs  that 
specify  the  robot  actions  necessary  to  achieve  the  desired  object  configurations. 
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In  task-level  languages,  the  program  specifies  the  functions  to  be  performed  or  tasks  to  be 
accomplished  (Lieberman,  1977).  The  ambition  is  to  have  language  constructs  at  the  level  of 
assembly  operations  such  as: 


insert  pegl  into  hole  a. 

Lower  levels  would  devise  a  grasping  strategy,  determine  error  recovery  methods,  and  plan  a 
collision  free  path  for  moving  the  peg. 

Sigr  'ficant  research  problems  must  be  solved  before  object-  and  task-level  programming 
languages  are  feasible.  Recent  attention  has  focused  on  the  study  of  supporting  functions  such 
as  grasp  selection,  error  avoidance,  error  recovery,  and  motion  planning  (Brooks, 
1982,Lozano-Perez,  1984,  Cutkosky,  1985).  While  considerable  progress  has  been  achieved  in 
understanding  these  subproblems,  little  eflfort  has  been  made  to  integrate  this  knowledge  into 
working  robot  systems.  If  high-level  languages  are  to  be  built  on  robot-level  languages,  then 
these  languages  must  provide  a  suitable  foundation  for  large  scale  software  development. 
Unfortunately,  robot-level  languages  have  not  been  designed  with  this  purpose  in  mind.  The 
tools  of  software  engineering  are  missing  from  most  of  these  languages.  Mechanisms  for  user 
defined  data  types,  information  hiding,  and  separate  compilation  are  unavailable  in  most 
robot-level  languages.  In  many  languages,  modularization  can  only  be  accomplished  with 
macro  substitution. 

We  chose  to  build  our  system  on  the  substrate  of  a  general  purpose  programming 
language  augmented  with  operations  for  controlling  the  robot  and  acquiring  images.  Programs 
were  written  in  the  C  language  on  a  Sun  color  workstation  running  the  Unix  operating  system. 
This  environment  provided  us  with  good  editing  facilities,  graphic  and  image  display  capabil¬ 
ity,  debugging  tools,  and  variety  of  helpful  utilities.  Our  work  relied  heavily  on  these  features. 

4.  A  Parametric  Language  for  Knot  Programming 

The  rope  was  abstractly  viewed  as  a  graph.  The  edges  of  the  graph  represent  uncrossed 
segments  of  rope.  The  vertices  of  the  graph  correspond  to  a  self-crossing  or  an  end  of  the  rope. 
Four  segments  are  incident  on  all  vertices  that  arise  from  a  crossing.  Since  a  single  strand  of 
rope  was  used,  the  edges  could  be  ordered  from  one  end  of  the  rope  to  the  other.  Following  the 
convention  of  knot  tying  books,  one  end  of  the  rope  is  designated  thc/ree  end  and  the  other  the 
standing  end  (Ashley,  1944,  Day,  1947). 

All  knot  commands  address  the  rope  by  its  graph  structure.  References  are  given 
parametrically.  For  example,  a  point  is  specified  by  a  segment  and  fraction.  When  a  command 
is  executed,  a  point  that  divides  the  coircsponding  segment  of  the  physical  rope  in  proportion 
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to  the  predetermined  fraction  is  located. 

A  graph  representation  of  the  rope  is  created  with  a  vision  system.  The  contour  of  the 
rope,  in  the  image,  is  attached  to  the  edges  of  the  graph.  Positions  and  orientations  on  the  rope 
are  mapped  into  robot  coordinates  when  arm  motion  commands  are  generated. 

Knot  commands  prescribe  a  place  to  grasp  the  rope  and  a  path  along  which  to  move  the 
rope.  The  grasp  location  is  specified  as  a  proportion  of  a  segment  The  destination  is  defined 
with  respect  to  some  other  point  on  the  rope.  An  intermediate  point  constrains  the  path  fol¬ 
lowed  to  reach  the  destination  point.  This  point  is  defined  as  a  deviation  from  the  straightline 
path.  Orientations,  as  well  as  positions,  are  always  defined  with  respect  to  the  orientaticn  of 
the  rop)e  at  some  point.  Thus,  all  references  are  bound  to  the  structure  of  the  rope  however  it 
happens  to  lie  at  the  time  the  command  is  executed. 

The  knot  da»a  type  maps  parametric  knot  operations  to  an  instance  of  a  physical  knot  and 
sends  motion  commands  over  an  asynchronous  communications  channel  to  the  Puma  con¬ 
troller.  The  data  type  also  assumes  responsibility  for  determining  grasp  orientation  and  wrist 
motion.  Early  experiments  demonstrated  that  the  rope  could  unpredictably  flip  if  signific  .itly 
twisted.  To  avoid  this  problem,  the  orientation  of  the  rope  held  in  the  hand  must  smoothly  fol¬ 
low  the  hand  trajectory.  The  rope  was  grasped  by  two  opposing  fingers.  The  parallel  grippers 
could  be  in  either  of  two  orientations  that  differed  by  180  degrees.  Since  the  robot  hand  has 
limited  range  of  wrist  motion,  care  had  to  be  taken  to  grasp  the  rope  with  the  maximum  range 
of  wrist  motion  in  the  direction  to  be  taken.  If  there  still  was  insufBcient  range  of  motion  to 
accomplish  the  wrist  rotation,  then  the  rope  was  set  down  midway  through  the  motion  and  the 
wrist  was  reversed- 

4.1.  Knot  Operations 

The  syntax  of  low  level  knot  commands  is  described  in  this  section. 

4.1.1.  Positions  and  Frames 

A  position  specifies  a  point  on  a  segment  of  the  rope.  Positions  are  defined  relative  to  the 
structure  and  shape  of  the  rope  at  execution  time.  The  segments  of  the  rope  are  numbered  from 
0  to  N,  beginning  at  the  standing  end.  A  segment  jwsition  refers  to  a  point  on  a  segment  as  a 
fraction  along  the  segment’s  length: 

<segment position  >::=  <segment  number  >  <proportion  > 

where, 

0  ^  <segment  number  >  <  number  of  segments 


and 
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0  ^  <proportion  >  ^  1. 

Points  not  on  the  rope  arc  defined  relative  to  a  segment _position.  A  local  coordinate 
frame  is  placed  on  a  segment jjosition  and  a  displacement  vector  is  specified  in  polar  coordi¬ 
nates: 

<relative position  >::=  <segment _position  >  <displacement  > 

where, 

Kdisplacement  >::=  <distance  >  <0>. 

We  assume  that  -7i<0,  <t><Jt  and  distance  ^0.  Distances  arc  measured  in  units  equal  to  the 
length  of  the  segment  in  the  defining  <segment _position  >. 

When  specifying  a  desired  placement  for  a  section  of  the  rope,  both  the  position  and 
orientation  of  the  rope  must  be  defined.  A  frame  combines  position  and  orientation  informa¬ 
tion: 

<frame  >::=  <relative _position  >  <^>. 

A  <frame  >  is  used  to  define  a  goal  position  and  orientation  with  respect  to  a  parametrically 
defined  point  on  the  rope. 

4.1.2.  Paths 

A  path  constrains  how  a  section  of  of  rope  will  be  moved.  Consider  a  point  on  the  rope 
with  position  Pq  and  orientation  4>o  in  an  arbitrary  coordinate  system.  Let  the  goal  destination 
for  a  move  operation  be  a  frame  with  location  P  i  and  orientation  <j>i .  The  trajectory  of  the 
hand  from  the  initial  to  goal  states  is  determined  by  a  single  intermediate  point,  P^,  through 
which  the  rope  point  must  pass.  The  point  P^  is  specified  by  a  single  signed  value,  the  excur¬ 
sion.  The  excursion  specifies  the  deviation  from  a  straight  line  path  from  the  initial  to  goal 
points.  It  is  defined  in  a  coordinate  frame  that  has  an  origin  at  the  midpoint  of  PoP\,  is 
oriented  in  the  direction  of  PqP\,  and  has  units  IIPo^iH-  A  vector  of  length 
excursion  •  II  i  II  is  placed  at  the  midpoint  ofPoP\  oriented  at  n/2  to  PqP  i  if  the  sign  of  the 
excursion  is  positive  and  at  -iU2  ioPqPi  if  the  sign  of  the  excursion  is  negative.  Joint  interpo¬ 
lated  motion  is  used  to  move  the  robot  from  Pq  to  P„  to  the  goal  destination  P  i . 

A  second  parameter  is  needed  to  specify  whether  the  orientation  of  the  rope  will  be 
rotated  in  a  clockwise  or  counterclockwise  direction  from  <|>o  to  <t>i.  Thus,  the  complete 
specification  of  a  path  is  given  by: 


<path  >:;=  <excursion  xdirection  > 
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where, 

<direction  >::=  clockwise  I  counterclockwise  I  min-angle. 

If  <direction>  is  min-angle,  then  the  rope  is  rotated  through  the  minimum  angle  to  get  from  <|)o 
to(t)i. 


4.13.  Operations 

The  language  includes  four  commands: 
grasp  <segment _posinon>  —  Grasp  the  rope  at  segment  j)osition. 

move  <frame>  <path>  —  Move  from  the  current  position  and  orientation  to  the  position  and 
orientation  specified  by  frame  along  a  route  specified  by  path. 
drop  —  Drop  the  rope  at  the  current  position. 

create  —  Create  a  new  representation  of  the  knot  with  the  vision  system. 

An  example  with  two  commands  and  the  geometric  interpretation  of  the  positions  and  paths 
defined  by  the  commands  is  presented  in  figure  2. 

KlhlSERT  FIGURE  2  ABOUT  HERE> 


4.1.4.  An  Example:  The  Figure  Eight  Kntrt 

A  portion  of  a  program  for  creating  a  figure  eight  knot  is  shown  in  figure  3.  Before  this 
segment  is  executed,  the  rope  will  have  been  configured  as  in  figure  2.  This  segment  forms  an 
eight  shape.  First,  the  free  end  is  crossed  over  the  base  of  the  rope.  The  second  move  state¬ 
ment  loops  the  firee  end  back  towards  the  standing  end,  where  it  is  dropped.  Lastly,  the  stand¬ 
ing  end  is  grasped  and  carried  over  the  free  end. 

grasp  0 .95 

move  0.35  .10-135-90.25  counterclockwise 

move  0.1  -.05  90  90.15  clockwise 

drop 

create 

grasp  0 .5 

move  0.5  -.28  90  0  0  min-angle 
drop 

Figure  3.  A  portion  of  the  program  for  tying  a  figure  eight  knot. 
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4.2.  The  Vision  System 

The  vision  system  provides  a  description  of  the  physical  rope.  This  descnption  binds  an 
abstract  representation  composed  of  segments  and  crossings  to  three-dimensional  contours 
along  the  skeleton  of  the  rope.  An  image  from  the  overhead  camera  is  thresholded  to  create  a 
binary  image  of  rope  and  background.  A  knot  structure  is  generated  by  following  the  skeleton 
of  the  rope  from  the  standing  end  to  the  free  end.  A  example  of  the  camera’s  view  of  the  rope 
is  given  in  figure  4. 


kINSERT  figure  4  ABOUT  HERE> 


First,  a  point  on  the  initial  segment  is  found  by  searching  the  border  of  the  image.  This 
can  be  done  because  the  the  rope  trails  off  the  end  of  the  table  and  out  of  view.  A  point  on  the 
opposite  side  of  the.  rope  is  located  by  searching  perpendicular  to  the  edge  of  the  rope.  The 
skeleton  of  a  segment  is  determined  by  parallel  tracking  of  opposite  sides  on  the  rope  cross- 
section.  Edge  points  are  iteratively  advanced  along  the  borders  of  the  rope  until  the  separation 
between  the  points  is  significantly  larger  than  the  expected  width  of  the  rope.  On  each  itera¬ 
tion,  the  line  segment  between  the  edge  points  is  compared  to  the  direction  of  the  midline  con¬ 
tour.  The  lagging  edge  point,  as  determined  by  the  angle  between  the  connecting  segment  and 
the  contour  direction,  was  always  advanced.  The  effect  is  that  the  two  points  shimmy  along  the 
rope  boundary,  siidmg  an  approximately  perpendicular  cross-section  along  the  rope  contour. 

The  rope  crossings  present  some  difSculty.  The  flexibility  of  the  rope  precludes  projec¬ 
tion  of  the  midline  across  the  crossing;  when  the  crossing  segments  are  nearly  parallel  the  likel¬ 
ihood  of  traversing  the  wrong  segment  is  high.  Instead,  the  crossing  is  mapped  by  following 
the  crossing  segments.  The  opposite  sides  of  the  two  crossing  segments  are  then  followed  back 
to  the  continuation  of  the  crossed  segment. 

This  process  is  repeated  until  the  end  of  the  rope  is  found.  A  knot  graph  is  constructed  as 
the  rope  is  traversed  and  skeletal  contours  are  attached  to  graph  edges.  The  contours  found  by 
the  vision  system  for  an  intermediate  step  in  tying  the  figure  eight  knot  are  shown  in  figure  4. 

5.  Graphic  Editing  and  Debugging 

Robot-level  programming  languages  have  been  frequently  criticized  for  requiring  pro¬ 
grammers  to  provide  numeric  specifications  of  three-dimensional  positions  and  orientations. 
These  measurements  are  difficult  for  programmers  to  originally  derive  and  are  later  difficult  for 
programmers  to  understand  and  modify.  A  number  of  robot-level  languages  have  been  aug¬ 
mented  with  physical  positioning  devices  to  simplify  coordinate  frame  definition  (Gini, 
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1985,Latombe,  1985). 

We  found  that  knot  programming  commands  were  also  cumbersome  to  write  and  difficult 
to  understand  once  written.  Software  was  developed  to  allow  knot  commands  to  be  graphically 
defined  and  displayed. 

A  graphic  tool  permits  a  programmer  to  interactively  step  the  robot  through  a  series  of 
knot  commands  specifying  robot  motions  by  pointing  with  a  cursor  on  an  image  of  the  rope. 
The  programmer  can  select  one  of  the  four  knot  commands  (grasp,  move,  drop,  or  create)  in  a 
text  window.  When  a  command  requires  geometric  arguments,  the  user  is  prompted  to  position 
the  cursor  on  the  image  and  click  the  mouse  button.  For  example,  the  move  command  requests 
in  succession  a  destination  point,  a  second  point  to  specify  the  desire^  orientation  of  the  rope  at 
the  destination,  and  a  mid-point  to  determine  the  path  excursion.  The  user  is  last  asked  to 
choose  the  direction  in  which  to  rotate  the  rope  from  the  i.nitiaJ  to  goal  orientation. 

Following  each  command  specification,  the  operation  is  executed.  A  create  command 
causes  the  image  to  be  updated  and  a  new  representation  of  the  knot  to  be  created.  For  move¬ 
ment  commands,  the  robot  executes  the  prescribed  motion  by  grasping,  moving,  or  dropping 
the  rope.  The  system  also  synthesizes  a  parametric  knot  command  from  the  absolute  positions 
and  orientations  chosen  by  the  user.  Each  parametric  command  is  printed  on  the  screen  as  it  is 
synthesized.  Given  an  exemplar  sequence  of  rope  manipulations,  a  general  program  can  be 
assembled. 

Figure  5  illustrates  the  graphic  interface  after  generation  of  a  grasp  and  move  command 
sequence. 


kINSERT  figure  5  ABOUT  HERE> 

A  create  command  caused  the  system  to  generate  a  up-to-date  description  of  the  rope.  Next, 
the  free  end  of  the  rope  was  grasped.  The  user’s  choice  for  the  grasp  point  is  marked  by  a  filled 
square  near  the  end  of  the  rope.  The  system  determined  the  nearest  point  on  the  rope  contour, 
shown  as  an  open  square.  The  contour  point  was  translated  into  a  scaled  length  along  the  struc¬ 
turally  ordered  segment  to  form  the  parametric  grasp  command  appearing  in  the  text  window. 

The  user  then  directed  the  system  to  move  across  the  the  first  segment.  The  two  points 
entered  to  determine  the  goal  position  and  orientation  are  indicated  by  a  filled  square  located  at 
the  first  selection  and  a  line  segment  connecting  the  first  and  second  selections.  The  graphi¬ 
cally  specified  destination  and  orientation  were  mapped  into  a  relative  position  and  orientation 
with  respect  to  a  nearby  point  on  a  rope  segment  The  reference  point  is  indicated  by  an  open 
square.  The  user  selected  one  additional  point  to  determine  the  excursion  parameter  for  the 
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mo'  c  and  the  direction  of  rotation.  Lastly,  the  constructed  move  command  was  printed  in  the 
lext  window. 

The  graphic  display  was  also  very  helpful  for  debugging  system  software.  When  robot 
commands  were  added  or  modified,  they  first  were  tested  by  projecting  the  motions  into  image 
space  and  drawing  the  result  on  the  display.  Gross  errors  were  detected  without  risk  of  damage 
to  the  robot. 

6.  Parser  and  Compiler  Generation  for  Higher-Level  Languages 

A  second,  more  abstract  layer  of  knot  tying  operations  was  built  on  top  of  the  p^jamctric 
language.  This  language  raises  references  fiom  the  level  of  points  and  paths  to  the  level  of 
segments  and  segment  crossings.  A  sequence  of  operations  in  the  parametric  language  is  com¬ 
piled  from  a  single  segment  command. 

The  segment-level  language  was  designed  to  allow  symbolic  names  to  be  attached  to  parts 
of  the  knot.  In  the  low-level  language,  all  references  to  a  knot  are  expressed  in  terms  of  the 
structure  of  the  rope.  Length  and  position  expressions  arc  bound  to  the  physical  rope  as  it 
exists  the  moment  the  reference  is  made.  The  physical  meaning  of  a  reference  can  change  as 
the  rope  is  manipulated.  For  example,  the  Nth  segment  will  become  the  N  +  Ur  segment  when¬ 
ever  a  segment  O.jV-1  is  crossed.  Small  modifications  to  the  object  can  have  global  conse¬ 
quences  for  the  referencing  scheme.  This  is  inconvenient  because  the  programmer  must  do  a 
great  deal  of  bookkeeping  to  keep  track  of  a  particular  piece  of  rope.  By  attaching  labels  to 
important  entities,  such  as  segments,  the  programmer  can  persistently  refer  to  a  structural  ele¬ 
ment  by  name.  In  the  segment-level  language,  names  persistently  refer  to  the  same  logical  seg¬ 
ment. 

The  fundamental  command  for  structuring  segments  is  cross.  A  cross  statement  directs  a 
crossing  of  the  last  segment  (the  segment  that  includes  the  free  end)  over  or  under  another  seg¬ 
ment  The  cross  command  optionally  allows  names  to  be  attached  to  new  segments  as  they  are 
created. 

Using  the  cross  command  the  end  of  the  rope  can  be  interwoven  through  the  knot.  This 
provides  a  simple  means  to  tie  many  knots.  The  syntax  for  cross  is: 

cross  {over  |  under} 

<seg-name>  from  {right  |  left} 

[curving  [widely  |  sharply]  [right  |  left}  ] 

[creating  <endname>] 

[splitting  <leftname>  <rightname>  ]. 
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Thc  elements  in  curly  brackets  are  mandatory  —  one  must  be  chosen.  The  elements  in  square 
brackets  are  optional.  This  command  causes  the  last  segment  to  cross  over  or  under  segment 
<seg-name>.  The  direction  of  the  crossing  is  determined  by  the  parameter  following  the  from 
term.  The  left  |  right  orientation  is  defined  for  someone  standing  on  the  rope  facing  the  free 
end.  The  optional  curving  element  guides  the  compiler  in  the  selection  of  the  path  to  the 
crossing. 

The  last  two  elements  associate  names  with  the  newly  created  segments.  Initially,  a  sin¬ 
gle  segment  named  "rope"  is  defined.  Each  cross  operation  can  create  up  to  three  new  named 
segments.  The  name  <endname>  in  the  creating  element  is  assigned  to  the  segment  that  starts 
at  the  new  crossing  and  continues  to  the  free  end.  The  splitting  element  assigns  names  <left- 
name>  and  <rightname>  to  the  left  and  right  segments  created  as  the  free  end  crosses  the 
<seg-name>.  The  names  are  illustrated  in  the  figure  6. 

KfNSERT  FIGURE  6  ABOUT  HERE> 

Names  continue  to  be  associated  with  the  length  of  rope  between  two  logical  crossings  as  the 
structure  of  the  is  modified  by  the  addition  of  new  crossings.  If  the  segment  is  split  by  a  subse¬ 
quent  crossing,  the  name  refers  to  set  of  segments  between  the  original  crossings. 

A  second  command  was  added  to  the  segment-level  language  to  allow  modification  of 
the  geometry  of  the  last  segment  The  extend  command  speci£"«  simple  shaping  and  enlarge¬ 
ment  of  the  last  segment  necessary  to  tie  certain  knots.  The  syntax  is 

extend  [a  little  |  a  lot] 

[curving  [widely  |  sharply]  (right  |  left }]. 

This  command  causes  the  last  segment  to  be  extended  and  possibly  turned  to  form  a  "J"  shape. 
The  modifying  parameters  a  little,  a  lot,  widely,  and  sharply  provide  rough  guidelines  for  the 
amount  of  extension  and  curvature  of  the  turn. 

The  language  is  best  understood  through  an  example  and  some  illustrations.  The  program 
for  a  figure  eight  knot  is  given  in  figure  7,  together  with  the  form  of  the  rope  and  the  named 
segments  created  by  each  command. 


<INSERT  FIGURE  7  ABOUT  HERE> 
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The  segment-level  language  was  implemented  using  the  Unix  utilities  yacc  and  lex.  A 
grammar  was  defined  and  then  converted  into  a  parser  and  compiler  for  the  language.  The 
compiler  makes  two  passes  over  a  knot  program.  In  the  first  pass,  a  logical  knot  is  created  by 
simulating  the  execution  of  the  knot  program.  An  abstract  representation  of  the  knot  is  built  as 
the  compiler  steps  through  the  program.  The  abstract  knot  has  the  topology  of  the  actual  knot 
but  is  attached  to  no  specific  contour.  In  the  second  pass,  the  compiler  adjusts  segment  lengths 
to  assure  that  when  the  segment  is  made  it  will  be  of  sufficient  length  to  permit  later  crossings. 
The  compiler  determines  specific  crossing  points  and  segment  lengths. 

For  the  sake  of  concreteness,  an  overview  of  the  compilation  of  a  cross  over  command  is 
presented  below. 

(1)  The  parser  recognizes  the  cross  over  tokens,  parses,  the  command,  and  returns 
tokens  for  each  of  the  potential  clauses:  segment,  fromdirection,  curve,  create,  and 
split. 

(2)  Low-level  comands  to  capture  an  image  and  grasp  the  free -end  of  the  rope  are 
assembled  to  form  a  skeletal  sequence  of  low-level  commands. 

(3)  The  move  command  that  the  will  do  the  actual  work  of  the  crossing  is  appended  to 
the  command  skeleton.  The  parameters  of  the  move  are  determined  from  the  seg¬ 
ment,  fromdirection,  and  curve  clause  information. 

(4)  A  crossing  is  created  in  the  abstract  knot  using  the  naming  and  direction  information 
in  the  clauses. 

(5)  A  drop  command  is  added  to  the  skeleton. 

The  basic  structure  of  the  low-level  command  sequence  i''  determined  by  these  five  steps. 
However,  all  parameters  are  not  fixed  until  a  second  pass  is  made  over  the  complete  knot  pro¬ 
gram.  Prior  to  the  second  pass,  the  final  abstract  knot  is  inspected  and  segments  are  assigned 
equal  lengths.  During  the  second  pass,  parametric  positions  are  assigned  in  the  grasp  and  move 
commands  based  on  the  desired  final  configuration.  At  this  point,  the  knot  program  is  complete 
and  ready  to  execute. 

Four  low-level  commands  are  assembled  to  implement  a  single  cross  over  command,  A 
cross  under  requires  a  series  of  grasp,  move,  and  drop  sequences.  First  the  end  of  the  rope  is 
brought  near  segment  to  be  crossed  and  dropped.  The  segment  is  then  lifted  over  the  free  end 
and  finally  the  end  of  the  rope  is  pulled  under  the  segment  to  the  destination  position.  In  other 
respects,  compilation  of  a  cross  under  statement  proceeds  as  above. 
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7.  Summary 

The  knot  tying  system  was  used  to  construct  programs  for  a  variety  of  common  knots. 
Knot  programs  were  tested  with  1/4  and  1/2  inch  width  nylon  ropes.  Initially,  the  unknotted 
rope  was  placed  in  a  random  orientation  near  the  center  of  the  workspace.  Experiments 
demonstrated  that  modestly  complex  knots  could  be  robustly  tied.  The  most  complicated  knot 
successfully  programmed  was  a  bowline  requiring  6  crossings. 

The  current  implementation  has  a  weak  error  detection  mechanism  and  no  error  correc¬ 
tion  capability.  Parametric  motions  adapt  to  the  rope  with  indifference  to  form  or  arrangement. 
A  command  that  references  the  nth  segment  is  valid  if  there  are  at  least  n  segments  in  the  knot, 
if  the  referenced  segment  does  not  exist,  then  an  error  message  is  printed  and  the  program  ter- 
minatej.  Little  more  error  detection  can  be  done  with  the  information  provided  by  parametric 
commands.  The  program  makes  no  explicit  statements  about  the  desired  structure  or  shape  of 
the  rope. 

The  segment-level  language  affords  a  greater  opportunity  for  error  analysis.  The  intended 
topology  is  direcdy  expressed  in  the  cross  commands.  Unsuccessful  crossing  operations, 
incidental  undoing  of  prior  crossings,  and  unplanned  crossings  can  lead  to  structural  abnormal¬ 
ities.  These  errors  could  be  detected  by  periodically  comparing  the  topology  of  the  knot  graph 
obtained  with  the  vision  system  to  the  topological  structure  implied  by  the  series  of  crossing 
commands.  During  compilation,  the  present  system  incrementally  constmcts  an  ideal  knot  by 
simulating  the  action  of  segment-level  commands.  This  abstract  model  has  the  topological 
structure  of  the  intended  knot  but  the  contours  are  absent  However,  this  information  is  not 
retained  in  the  compiled  program.  Error  checking  could  be  introduced  by  interpreting,  rather 
than  compiling  the  segment  level  language.  Error  correction  raises  a  number  of  difficult  plan¬ 
ning  problems  beyond  the  scope  of  the  present  system. 

Neither  of  the  current  languages  explicitly  describe  the  geometry  of  the  rope.  For  simple 
knots,  the  two-dimensional  topology  of  the  rope  provides  sufficient  constraints  to  accomplish 
the  task.  But  for  more  con^plicated  knots,  the  geometry  of  uncrossed  segments  is  also  impor¬ 
tant.  Most  knots  can  be  tied  in  a  variety  of  ways.  Many  common  techniques  require  that  seg¬ 
ments  of  the  rope  be  formed  into  shapes  in  preparation  for  future  operations  (Ashley, 
1944,  Day,  1947).  Figure  8  demonstrates  two  steps  in  tying  a  sheepshank. 

<INSERT  FIGURE  8  ABOUT  HERE> 

The  rope  is  first  formed  into  an  elongated  "S"  shape.  More  global  geometric  descriptions  are 
needed  to  characterize  some  complex  knots.  For  example,  it  is  important  that  the  three  loops  in 
figure  9  be  colinear  and  approximately  the  same  size.  An  ability  to  characterize  the  important 
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<INSERT  FIGURE  9  ABOUT  HERE> 

geometric  properties  of  segments  and  relationships  between  parts  of  the  knot  is  needed  to 
determine  the  success  or  failure  of  manipulations  with  intricate  knots.  The  design  of  shape 
representations  adequate  to  express  the  qualitative  properties  of  complex,  self-intersecting 
figures  is  a  challenging  problem. 

The  purpose  of  this  project  was  to  explore  programming  of  physical  operations  on  objects 
where  it  is  impossible  to  precisely  specify  the  actions  to  be  performed.  The  knot  ty’  .  study 
demonstrated  the  difficulties  that  arise  in  constructing  robust,  non-metric  robot  programs.  In 
working  on  the  project,  the  fundamental  importance  of  two  notions  became  apparent:  program¬ 
ming  on  abstractions,  and  the  interrelated  roles  of  language,  action,  and  perception  in  doing  so. 
Appropriate  abstractions  allow  the  programmer  to  address  the  properties  of  the  object,  not  the 
actions  of  the  robot.  Parametric  motion  commands  are  step  towards  more  qualitative 
languages.  The  experience  also  emphasized  how  programming  tools  can  contribute  to  building 
robot  programming  systems. 


Acknowledgments 

This  research  was  supported  in  part  by  National  Science  Foundation  Grants  DCR85- 
02568,  DCR86-17355,  and  IRI-8808896  and  Ofifce  of  Naval  Research  Grants  ONR  00014- 
86K-0281  and  N0014-88-K-0632.  We  would  like  to  thank  Douglas  Cambell  for  his  valuable 
assistance. 


-  16- 


References 

1.  Ambler,  A.,  Cameron,  S.  A.,  and  Comer,  D.  F.,  “Augmenting  the  RAPT  robot  language,’’ 
in  Languages  for  Sensor-based  Control  in  Robotics,  ed.  Hormann,  pp.  305-316,  Springer- 
Verlag,  New  York,  NY,  1987. 

2.  Ashley,  C.W.,  The  Ashley  Book  of  Knots,  Doubleday,  Doran  and  Company,  Inc.,  Garden 
City,  New  York,  1944. 

3.  Brooks,  R.,  “Symbolic  error  analysis  and  robot  planning,”  The  International  Journal  of 
Robotics  Research,  vol.  1,  no.  4,  pp.  29-68,  Winter  1982. 

4.  Cutkosky,  M.  R.,  Robot  Grasping  and  Fine  Manipulation,  Kluwer  Academic  Publishers, 
Boston,  MA,  1985. 

5.  Day,  C.L.,  The  Art  of  Knotting  and  Splicing,  Dodd,  Mead  and  Company,  Inc.,  New  York, 
New  York,  1947. 

6.  Finkel,  R.,  Taylor,  R.,  Bolles,  R.,  Paul,  R.  P.,  and  Feldman,  J.,  “An  overview  of  AL:  A 
programming  system  for  automation,”  Fourth  International  Joint  Conference  on  Artificial 
Intelligence,  pp.  758-765, 1975. 

7.  Geschke,  C.  C.,  “A  system  for  programming  and  controlling  sensor-based  robot  manipu¬ 
lators,”  IEEE  Transactions  on  Pattern  Analysis  and  Machinge  Intelligence,  vol.  PAMI-5, 
no.  1,  pp.  1-7,  Jan  1983. 

8.  Gini,  G.  and  Gini,  M.,  “Dealing  with  world-model-based  programs,”  ACM  Transactions 
on  Programming  Languages  and  Systems,  vol.  7,  no.  2,  pp.  334-347,  April  1985. 

9.  Inoue,  H.  and  Inaba,  H.,  “Hand-eye  coordination  in  rope  handling,”  in  Robotics 
Research:  The  First  International  Symposium,  ed.  R.  Paul,  pp.  163-174,  MIT  Press,  Cam¬ 
bridge,  MA,  1984. 

10.  Latombc,  J.  C.,  Laugier,  C.,  Lefebvre,  J.  M.,  Mazer,  E.,  and  Miribel,  J.  F.,  “The  LM  robot 
programming  system,”  in  Robotics  Research  -  The  Second  International  Symposium,  ed. 
H.  Inoue,  pp.  377-391,  MIT  Press,  Cambridge,  MA,  1985. 

11.  Lieberman,  L.  I.  and  Wesley,  M.  A.,  “AUTOPASS:  An  automatic  programming  system 
for  computer  controlled  mechanical  assembly,”  IBM  Journal  of  Research  and  Develop¬ 
ment,  vol.  21,  no.  4,  pp.  321-333, 1977. 

12.  Liskov,  B.  and  Guttag,  J.,  Abstraction  and  Specification  in  Program  Development,  MIT 
Press,  Cambridge,  MA,  1986. 

13.  Lozano-Perez,  T,  “Robot  Programming,”  Proceedings  of  the  IEEE,  vol.  71,  no.  7,  July, 
1983. 


-  17- 


14.  Lozano-Perez,  T.,  Mason,  M.,  and  Taylor,  R.,  “Automatic  synthesis  of  fine-motion  stra¬ 
tegies  for  robots,’’  The  International  Journal  of  Robotics  Research,  vol.  3,  no.  1,  pp.  3-24, 
Spring  1984. 

15.  Paul,  R.,  “WAVE;  A  model-based  language  for  manipulator  control,”  Technical  Paper 
MR  76-615,  pp.  10-17,  Society  of  Manufacturing  Engineers,  1976. 

16.  Poplestone,  R.,  Ambler,  A,  and  Bellos,  I.,  “An  interpreter  for  a  language  for  describing 
assemblies  ”  Artificial  Intelligence,  vol.  14,  pp.  79-107,  1980. 

17.  Poplestone,  R.  and  Ambler,  A.,  “A  language  for  specifying  robot  manipulations,”  in 
Robotic  Technology,  ed.  A.  Pugh,  pp.  125-141,  Peter  Peregrinus,  London,  1983. 

18.  Shimano,  B.,  “VAL;  A  versatile  robot  programming  and  control  system,”  COMPSAC  79 
Conference  Proceedings,  pp.  878-883, 1979. 

19.  Taylor,  R.,  Summers,  P.,  and  Meyer,  J.,  “AML:  A  manufacturing  language,”  The  Inter¬ 
national  Journal  of  Robotics  Research,  vol.  1,  no.  3,  pp.  19-41,  Fall  1982. 

20.  Volz,  R.,  “Report  of  the  robot  programming  language  working  group:  NATO  workshop 
on  robot  programming  languages,”  IEEE  Journal  of  Robotics  and  Automation,  vol.  4,  no. 
l.pp.  86-90,  Feb  1988. 


Figure  1.  The  robot  performing  an  intermediate  step  of  a  knot  program.  A  segment  is  being 
lifted  over  the  free  end. 
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Figure  2.  Two  commands  from  a  program  to  tie  a  figure  eight  knot  and  their  geometric 
interpretation  for  an  instance  of  the  rope. 


Figure  4.  The  contours  generated  by  the  vision  sytem  are  drawn  on  the  rope  image. 
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Figure  5.  The  graphic  interface  for  specifying  knot  commands. 
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Figure  6.  Naming  conventions  for  the  segment  level  language 
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Figure  7.  The  state  of  the  rope  after  each  step  of  a  segment-level  program  to 
tie  a  figure  eight  knot. 


Figure  8.  Two  steps  in  tying  a  sheepshank  knot. 


Figure  9.  Two  steps  in  tying  a  masthead  knot. 


